Why Does Bagging Work? A Bayesian Account and its Implications

The error rate of decision-tree and other classi-cation learners can often be much reduced by bagging: learning multiple models from bootstrap samples of the database, and combining them by uniform voting. In this paper we empirically test two alternative explanations for this, both based on Bayesian learning theory: (1) bagging works because it is an approximation to the optimal procedure of Bayesian model averaging, with an appropriate implicit prior; (2) bagging works because it eeectively shifts the prior to a more appropriate region of model space. All the experimental evidence contradicts the rst hypothesis, and connrms the second. Bagging Bagging (Breiman 1996a) is a simple and eeective way to reduce the error rate of many classiication learning algorithms. For example, in the empirical study described below, it reduces the error of a decision-tree learner in 19 of 26 databases, by 4% on average. In the bagging procedure, given a training set of size s, a \bootstrap" replicate of it is constructed by taking s samples with replacement from the training set. Thus a new training set of the same size is produced, where each of the original examples may appear once, more than once, or not. On average, 63% of the original examples will appear in the bootstrap sample. The learning algorithm is then applied to this training set. This procedure is repeated m times, and the resulting m models are aggregated by uniform voting. Bagging is one of several \multiple model" approaches that have recently received much attention (see, for example , (Chan, Stolfo, & Wolpert 1996)). Other procedures of this type include boosting (Freund & Schapire 1996) and stacking (Wolpert 1992). Two related explanations have been proposed for bagging's success, both in a classical statistical framework. Breiman (1996a) relates bagging to the notion of an order-correct learner. A learner is order-correct for an example x if, given many diierent training sets, it predicts the correct class for x more often than any other. Breiman shows that, given suucient replicates, bagging turns an order-correct learner into a nearly-optimal one. Although this line of reasoning has intuitive value, its usefulness is limited, because it is seldom (or never) known a priori whether a learner is order-correct for a given example or not, or what regions of the instance space it will be order-correct in and not. Thus it is not possible to judge from an application domain's characteristics whether bagging …